Scenarios Analyses
2025-07-02
1. Overview of scenario data
First, I look at how participants rated the appropriateness of each violation in each scenario, and the appropriateness of the different types of punishments in that scenario.
Data is displayed separately by treatment, i.e. whether the violation is done by a friend/non-friend, and weather the punishment is performed by a friend to a friend/non-friend. Treatments are between subjects.
Insights:
All violations are considered fairly inappropriate, with insulting a
family member and stealing being the worst.
There is quite some variation in appropriateness of the different types
of punishment.
There are no striking differences between treatments.
Stealing
Phone
Interrupt
Insult
Avoid
1.1. Correlation of appropriantess of violations and appropriateness of punishment.
Hypothesis: more appropriate a behavior is, the less appropriate the punishment should be (neg corr), except doing nothing (pos corr)
2. Analysis of appropriateness of violations across all domains
Now I merge all the scenarios together.
Insight:
Overall appropriateness is quite low, with again no striking difference between treatments.
2.1 ANOVA
First I run an ANOVA to get an idea of differences across treatments and domains. The ANOVA showed that both treatment and domain significantly affected ratings, with a significant interaction indicating that the effect of treatment varied across domains.
## Df Sum Sq Mean Sq F value Pr(>F)
## treatment 1 12 12.50 9.080 0.00261 **
## domain 4 324 80.97 58.817 < 2e-16 ***
## treatment:domain 4 47 11.71 8.507 8.16e-07 ***
## Residuals 2433 3349 1.38
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 12.2 Regressions
I now run two multilevel linear regressions with participants as random intercept to zoom in on size and direction of efffects.
First, a linear regression predicting rating ~ treatment shows a small effect of treatment: violations by friends are considered slightly more appropriate.
If we include a test of interactions between treatment and domain, we can see that the effect is driven by the “exclude from online chat scenario:
“One student who is part of your friends/others group (Student A) excluded another student, who is also a part of your friends group, from a group chat on WhatsApp.”
Rating ~ Treatment
| rating | |||
|---|---|---|---|
| Predictors | Estimates | CI | p |
| (Intercept) | 1.08 | 1.00 – 1.16 | <0.001 |
| treatmentf | 0.14 | 0.05 – 0.23 | 0.003 |
| Random Effects | |||
| σ2 | 1.26 | ||
| τ00 ID2 | 0.27 | ||
| ICC | 0.18 | ||
| N ID2 | 496 | ||
| Observations | 2443 | ||
| Marginal R2 / Conditional R2 | 0.003 / 0.178 | ||
Rating ~ Treatment * Domain
| rating | |||
|---|---|---|---|
| Predictors | Estimates | CI | p |
| (Intercept) | 0.61 | 0.46 – 0.75 | <0.001 |
| treatmentf | -0.04 | -0.25 – 0.17 | 0.721 |
| domain interrupt | 0.53 | 0.32 – 0.74 | <0.001 |
| domain [online] | 0.68 | 0.50 – 0.87 | <0.001 |
| domain stealing | 0.35 | 0.14 – 0.56 | 0.001 |
| domain [texting] | 0.79 | 0.60 – 0.97 | <0.001 |
|
treatmentf × domain interrupt |
0.13 | -0.19 – 0.46 | 0.421 |
|
treatmentf × domain [online] |
0.74 | 0.48 – 1.00 | <0.001 |
|
treatmentf × domain stealing |
-0.01 | -0.33 – 0.32 | 0.970 |
|
treatmentf × domain [texting] |
0.07 | -0.19 – 0.33 | 0.602 |
| Random Effects | |||
| σ2 | 1.07 | ||
| τ00 ID2 | 0.31 | ||
| ICC | 0.22 | ||
| N ID2 | 496 | ||
| Observations | 2443 | ||
| Marginal R2 / Conditional R2 | 0.103 / 0.301 | ||
3. Analysis of appropriateness of punishment if friend vs. non-friend
Now I look at the appropriateness ratings of punishments if the violator is a friend vs. non friend.
3.1 Vizualization
Again, visual inspection does not reveal any striking differences bewteen treatments and domains.
Overall appropriatess across domains
Appropriateness by domain
Appropriateness of punishment by treatment and punisment type
3.1 ANOVA
First, I run an ANOVA to get an idea of differences in mean ratings across treatments and domains.
There was no significant main effect of treatment.
In contrast, domain was significant, suggesting that the ratings varied meaningfully across different domains.
The interaction between treatment and domain was also significant, implying that the effect of treatment differed depending on the domain.
## Df Sum Sq Mean Sq F value Pr(>F)
## treatment 1 0 0.00 0.002 0.9611
## domain 4 475 118.69 68.455 <2e-16 ***
## treatment:domain 4 15 3.83 2.208 0.0656 .
## Residuals 7284 12630 1.73
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 13.2 Regressions
Now I fit 3 linear regressions to better explore directions of effects.
First, only with treatment as independent variable, then I add interactions with domain (insult as reference) and finally I also include punishment type as exploratory variable (angry_remark as reference).
The main results are [see tabs below for full results]:
Model 1: no effect
Model2 :
- Raters considered more appropriate to punish friends who exclude others from group chats.
Model 3:
- The domain “texting” was associated with significantly lower ratings compared to the reference domain.
- The punishment types “do nothing” and “gossip” led to significantly lower ratings compared to the reference punishment type.
There a few significant interactions:
The combination of “interrupt” domain and “avoid” punishment type was associated with lower ratings compared to the reference groups.
The combination of “online” domain and “do nothing” punishment type was associated with lower ratings compared to the reference groups.
The combination of “stealing” domain and “do nothing” punishment type was associated with lower ratings compared to the reference groups.
The combination of “texting” domain and “do nothing” punishment type was associated with higher ratings compared to the reference groups.
The combination of “stealing” domain and “gossip” punishment type was also associated with higher ratings compared to the reference groups.
Rating ~ Treatment
| rating | |||
|---|---|---|---|
| Predictors | Estimates | CI | p |
| (Intercept) | 2.54 | 2.48 – 2.60 | <0.001 |
| treatmentf | -0.01 | -0.06 – 0.05 | 0.833 |
| Random Effects | |||
| σ2 | 1.56 | ||
| τ00 ID2 | 0.24 | ||
| ICC | 0.13 | ||
| N ID2 | 496 | ||
| Observations | 7294 | ||
| Marginal R2 / Conditional R2 | 0.000 / 0.131 | ||
Rating ~ Treatment * Domain
| rating | |||
|---|---|---|---|
| Predictors | Estimates | CI | p |
| (Intercept) | 2.61 | 2.50 – 2.72 | <0.001 |
| treatmentf | 0.01 | -0.15 – 0.16 | 0.936 |
| domain interrupt | -0.20 | -0.35 – -0.04 | 0.012 |
| domain [online] | -0.14 | -0.27 – -0.01 | 0.030 |
| domain stealing | 0.36 | 0.21 – 0.52 | <0.001 |
| domain [texting] | -0.41 | -0.53 – -0.28 | <0.001 |
|
treatmentf × domain interrupt |
-0.01 | -0.26 – 0.23 | 0.907 |
|
treatmentf × domain [online] |
0.16 | -0.02 – 0.34 | 0.078 |
|
treatmentf × domain stealing |
-0.09 | -0.33 – 0.16 | 0.491 |
|
treatmentf × domain [texting] |
-0.09 | -0.27 – 0.09 | 0.320 |
| Random Effects | |||
| σ2 | 1.49 | ||
| τ00 ID2 | 0.24 | ||
| ICC | 0.14 | ||
| N ID2 | 496 | ||
| Observations | 7294 | ||
| Marginal R2 / Conditional R2 | 0.037 / 0.171 | ||
Rating ~ Treatment * Domain* punishment type
| rating | |||
|---|---|---|---|
| Predictors | Estimates | CI | p |
| (Intercept) | 2.89 | 2.73 – 3.05 | <0.001 |
| treatmentf | -0.11 | -0.34 – 0.12 | 0.361 |
| domain interrupt | -0.11 | -0.34 – 0.12 | 0.355 |
| domain [online] | -0.16 | -0.38 – 0.05 | 0.131 |
| domain stealing | 0.14 | -0.09 – 0.37 | 0.231 |
| domain [texting] | -0.50 | -0.71 – -0.28 | <0.001 |
| punishment type avoid | -0.15 | -0.36 – 0.07 | 0.175 |
| punishment type [gossip] | -0.69 | -0.91 – -0.48 | <0.001 |
|
treatmentf × domain interrupt |
0.17 | -0.18 – 0.52 | 0.329 |
|
treatmentf × domain [online] |
0.27 | -0.04 – 0.57 | 0.084 |
|
treatmentf × domain stealing |
0.09 | -0.26 – 0.44 | 0.605 |
|
treatmentf × domain [texting] |
0.04 | -0.27 – 0.34 | 0.816 |
|
treatmentf × punishment type avoid |
0.17 | -0.13 – 0.47 | 0.264 |
|
treatmentf × punishment type [gossip] |
0.17 | -0.13 – 0.47 | 0.271 |
|
domain interrupt × punishment type avoid |
-0.41 | -0.71 – -0.11 | 0.007 |
|
domain [online] × punishment type avoid |
-0.08 | -0.38 – 0.22 | 0.614 |
|
domain stealing × punishment type avoid |
0.31 | 0.01 – 0.61 | 0.041 |
|
domain [texting] × punishment type avoid |
-0.04 | -0.34 – 0.26 | 0.813 |
|
domain interrupt × punishment type [gossip] |
0.15 | -0.15 – 0.45 | 0.325 |
|
domain [online] × punishment type [gossip] |
0.15 | -0.15 – 0.45 | 0.327 |
|
domain stealing × punishment type [gossip] |
0.36 | 0.06 – 0.66 | 0.018 |
|
domain [texting] × punishment type [gossip] |
0.30 | 0.00 – 0.60 | 0.050 |
|
(treatmentf × domain interrupt) × punishment type avoid |
-0.22 | -0.65 – 0.20 | 0.303 |
|
(treatmentf × domain [online]) × punishment type avoid |
-0.21 | -0.64 – 0.22 | 0.338 |
|
(treatmentf × domain stealing) × punishment type avoid |
-0.22 | -0.65 – 0.20 | 0.300 |
|
(treatmentf × domain [texting]) × punishment type avoid |
-0.10 | -0.52 – 0.33 | 0.661 |
|
(treatmentf × domain interrupt) × punishment type [gossip] |
-0.34 | -0.77 – 0.08 | 0.115 |
|
(treatmentf × domain [online]) × punishment type [gossip] |
-0.11 | -0.54 – 0.32 | 0.619 |
|
(treatmentf × domain stealing) × punishment type [gossip] |
-0.31 | -0.74 – 0.11 | 0.152 |
|
(treatmentf × domain [texting]) × punishment type [gossip] |
-0.28 | -0.71 – 0.14 | 0.195 |
| Random Effects | |||
| σ2 | 1.44 | ||
| τ00 ID2 | 0.24 | ||
| ICC | 0.15 | ||
| N ID2 | 496 | ||
| Observations | 7294 | ||
| Marginal R2 / Conditional R2 | 0.070 / 0.205 | ||